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ABSTRACT i , . 

The objectives of the project were to (1) determine 
whether students making a subjective evaluation of other students 
will tend to JB^^ their friends higher than others th^y do not know 
as well, (2) 'VLether the use of an evaluation instrument reduces the 
sub jectivitiy of student judgment, and (3) what differences there may 
be between a grade based on a general impression and one based on 
evaluating particular points. The research-instructor prepared an 
evaluation form for students to use in subjective grading of other 
students' performances on 12 points. The evaluators also furnished 
numerical scores of their overall impressions of the performances. It 
was found that students gave lower grades when using the evaluation^ 
instrument than on the general impression. Students also graded their 
friends higher by both methods. The study indicat^ad that a moDe 
realistic evaluation can be obtained by using an instrument requiring 
evaluation of several specific points rather than basing the grade on 
a general overall impression. It was concluded than when an 
instructor wants assistance in subjective evaluations he may use 
student" evaluations if an effective evaluation instrument is prepared 
and the students' evaluations are averaged with those of the 
instructor. Appendices include the instrument used, analysis sheets, 
summary of scores, and summary analysis sheet's. (MF) 
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' ..) . . statement of Problem 

When a student's performance must be measured subjectively, or when 
it is difficult to quantify, a better evaluation can usually be ob- 
tained if more than one person judges the presentation. Some teachers 
may feel that they are better qualified than anyone else to judge the 
student and assign a grade, but this attitude must be questioned. 

r 

^For example, if a panel of three were to judge a speech contest, one 
member might be influenced by appearance, another by content and 
another by style. With or without a rating instrument, which would 
evaluate these specific points, the evaluations would probably bal- 
ance out so that the highest grades would go, to the students giving 
the best performance unless all of the judges were>iased in the same 
direction. Obviously, a combined judgement should be superior to the 
judgement of an individual when an overall evaluation is desired. 

When presentations requiring subjective analysis are made contin- 
uously during a school year, a typical classroom instructor is faced 
with a problem. The instructor can do the evaluation alona or can 
attempt to obtain the assistance of other instructors or even persons 
outride of the institution. However, this assistance is ysually dif- 
ficult to obtain on a continuing basis. 

Another possibility is to have the students evaluate each other as 
well as having the performance evaluated by the instructor. The 
question than an>-s as to whether or not the students' appraisals 



'would be fair and honest. A reasonable assumption V/oiild be that the 
^.students' knowledqe of each other would unduly influence their judge- 
ment. Specifically, if , i^t could be assumed that they would tend to • 
favor their friends and close acquaintances and downgrade those whom 
they disliked or with whom.,they were not well acquainted. - 

In many cfasses, students are graded on how well they perform certain 
tasks and it may be difficult, if not impossible, to set up criteria 
that can be measured objectively. If a, subjective evaluation is nec- 
essary, the instructor may not wish tt be the sole judge, especially 
if he has had the students in other /lasses. His knowledge of their 
-behavior and abilities might influence his current evaluation. 

-If the instructor were- to permit the other members of the class to 
participate in the evaluations, the problem is thatthese student 

"evaluations might be too subjective and biased to be used in de- - 
termining the final grade. 



Objectives. 



The basic purpose of this project was trdetemine whether students, 
when making a subjective evaluation,. tend to grade their friends and 
close acquaintances higher than others they do not know as well. ^ 

A secondary objective was to determine whether the use of an evaluation 
instrument by students reduces the subjectivity of their judgement. 
Specifically, an attempt was made to determine the difference between 
a grade based on a general overall impression of the presentation and— 
one based on evaluating particular points. 

/ • • ■ 
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Background Information 

The reasearcher has, for four years, taught a course called Purchasing 
Negotiations which is part of the Materials Management Curricula at 
Lakeshore Technical Institute, Cleveland, Wiscon"?in. 



In |Kls course, students are trained in the negotiating skills needed 
by an industrial purchasing agent/" A hands-on approach is used by 
having the students resolve conflicts 'through negotiation. The first ; 
half of the course consists, in addition to some lectures, of having 
students develop a situation in which they, as a. representative of a 
manufacturer, have to negotiate a settlement with industrial buyers. 
The situations represent such typical problems as a price increase, . 
poor quality, late "del i very, etc. Other students are selected as the 
•buyers' representatives and a negotiating session is then privately 
video-taped. The students are given fifteen td thirty minutes during 
which time. they are to reach a mutually satisfactory agreanent. 

/ 

r 

students are required to provide the instructor privately with 
specific objectives they are trying to rea^h. For example, the man- 
ufacturer's representative may state that he is asking for an eighth 
percent, price increase across the board but will settle for five percent, 
and the buyer may state that he will offer a two percent price increase 
but will settle for four percent. Obviously, these goals are'incom- 
patiblelnd someone is going to have to make some concession^if an 
. agreanent is to be reached. Usually other fzctors such as freight 

— 8 ' y 

t • ( 
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absorption, packaqing, quality control specifications, and so forth are 
irtroduced so that one item may be^traded off" for another,. This, is a 
real life situation and some kind of an agreement is usually reached. 

When several sessions- are completed, the tapes are played, b|ck before 
the entire class for discussion and critique. 

Initially, the instructor was the sole judge and assigned a grade based 
on his general impression of their performance and how close the students 
came to achiei^ng their stated objective. During the 1975 spring term, 
the instructor prepared a formal evaluation instrument which students 
used to evaluate the negotiating sessions. The participants did not 
graire themselves. Other students' evaluations were averaged with the 
mean of student evaluations counting one-third and the instructor's^) 
evaluation counting two- thirds on the grade of that presentation. 

This is the background and the basis for the present investigation. 



9 



. PROCEDURE 

Before playinq back a video-taped negotiation session for a critique 
and evaluation by the class, studetits were given the "Evaluation of 
Purchasing Negotiation f'resentatio^n" form. Exhibit One. It was cex- 
p' ined that they were to grade the buyer and seller on 12 points, 
that these would be totaled and a simple average (mean) of all the 
student's evaluations calculated. This would then be averaged with the 
instructor's evaluations (50/50) to determine the grade on that partic- 
ular presentation. The 12 criteria on the form were discussed so that 
all students would be more aware of the points to watch for during a 
playback period. They were instructed to keep' notes on the back of the 
page^ and to complete the evaluation following a general critique. 

i 

^- ! , 

The 12 'criteria used for judging do not all have equal weight. Five have 
a possible, score of ten, five a possible score of eight, and two a pos- 
sible score of five; the total being 100. This instrument was used the 
previous year and was considered quite satisfactory. 

The video-tape was then played, and upon conclusion, the- students were 
asked to immediately give tr?e buyer and seller a numerical score from 
zero. to 100 based on their overall impression of the 'negotiating session 
and to. write it at the bottom of the columns on the evaluation sheet. A 
g,eneral critique foUlowed during which the instructor acted as the mod- 
erator but avoided making any positive or negative statements -cpncerning 
the participants. 
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The students than completed the evaluation instrument, folded i^t in 
half and gave it to the instructor. 



Originally there were 22^^students in the class e.nd each was to partic- 
ipate in two negotiating sessions. Due to drop ou'r.s, job outs, and 
absences during the critique, a total of 680 evaluations and 644 gen- 
eral impression scores were received. These were tabulated and the mean 
for the buyer and seller were calculated in order to determine a grade. 



For the purpose of this report, the mean of both evaluations, one from 
the evaluation sheet and one from the general impression grade that 
each student gave, were calculated. 

A^ter all of the negotiating sessions had been evaluated, the students 
were given a survey sheet. Exhibit Two. This su-vey asLed three questions 
regarding the relationship of the student to the other students in th^ 
class; 1) How much timeSo they spend together butside of class, 2) How < 
long have they known him, and 3) How well areft'hey acquainted. They . 
were told that the instructor was doing a research project and needed the 
information and it had nothing to do with their gr^ues and the^course. 
It was explained that the information received would be kept confidential 
and be seen only by the instructor, and if they felt it was too personal, 
they had no obligation -to complete the form. However, all of the students 
in the class completed the survey. 

This con^pleted the information gathering portion" of the project. 



Organization and Presentation of Data' 



After each video-tape session was evaluated, the scores were posted- to two 
tally sheets, one for the scores fran the evaluation instrument and one 
for the general impression scores. These tally sheets aise'.not included in 
this report because they contained personal ^ames in^^rder to facilitate ■ 
>6sting from the evaluation. sheets which also used personal names. When all 
of the evaluations were complete, the mean of the scores each student gave 
was calculated. 



The st.udents' survey sheets were th^n analyzed. In order to obtain a numer- 
ical score for comparison purposes, values had to be assigned to the re- 
sponses!^ Values Were assigned as shown. 
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It was decided th^ lowest total value possible would be zero, and the highest 
value would be ten. Because the purpose of the research is to determine 
if students will grade their close friends and those with whom they are 
well acquainted higher than those they do not know as well, the -response 
"a ve>ty close friend" was given a value of four. The assumption was 2 
that the students are likely to be more friendly with someone they spend 
five hours per week with than someone they nleyer see outside of class; 
therefore, the response "more than five hours" was given a value of three. 
Finally, it was assumed that the longer they have known someone, the more 
likely they are to be friends; therefore, the response "more than two years" 
was given a value of three. 

It is recognized that these assumptions wiltnot hold true in every case. 
A person could know someone more than two years and totally dislike him or 
a person could be in contact with another for more than five hours and still 
not consider him a friend. However, when the values checked for each question 
are totaled, it does seem reasonable to assume that the student will be 
closer to those with the higher value total than to those with the lower 
total. It is uDon this assumption that the analysis of the results are 
based. 

The response valusz across were totaled as in the example and the three 
highest and three lowest totals for each evaluator noted. These totals 
were posted to the "f" column of an analysis sh^t. Exhibit Three. The 
evaluator.'s I.D. number v-^as recorded on the top Vine along with the mean 
'm" of each of his evaluations, one from the evaluation instrument and 
one from his general impression scores. The I.D. numbers and the scores 
given to six students, three'with the h'^ghest total values and the three 



lowest total values, were posted. The mean "m" of each student's given 
scores was calculated, and the variance "v" from the mean of the eval- 
uator's scores was posted,.. T-scores were not used because they would 
not provide any better comparisons than the variances between the means 
of the raw scores. The mean and variance for the combined scores for 
the top and bottom three were also calculated, and this was used to 
make the final comparison. Variance for the combined scores is circled 
on each analysis sheet. 

When the analysis sheets were completed, a summary sheet was prepared. 
Exhibit Four. The evaluator's mean scores using the evaluation instru- 
ments were listed in descending order in one column, and" the mean general 
Impression scores from the same student were listed in another, column. 
The difference "d" between the variance for the three students he 
"knows best" and the three students he "knows least" was posted next to 
the mean score. For example, a -5 indicates that he graded the three 
he "knows least" an average of five points lower than the three he "knew 
best." 

Another column was added to this summary sheet to show the difference 
between the mean raw scores from the evaluation instrument and from the 
general impression scores. 

For example, student No. 8837 gave an average score of 97 using the 
evaluation instrument and an average general impression score of 95 for 
a difference of negative two points. Whereas student 5678 did just the 
opposite and gave an average general impression score of 87, 14 points 
higher than the average score of 73 he gave using the evaluation instru- 
ment. ' It is interesting to note that students giving the lower average 
scores from the evaluation instrument generally gave considerable higher 
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AMALYSIS 



^ ' A studjof the "Sumnary of Analysis Sheets," Exhibit Four, reveals sev- 

. era! points worth consi deration. 

/ 

\ 

When comparing the grades the students gave using the evaluation instru- 
ment to the general impression grades, it is apparent that they g-.ve 
lower grades when using the evaluation instrument. The average grade 
from the evaluation instrument was 87.5 while the average grade from the 
general impression score was 91.1, a difference of 3.6 points. Equally 
important was the range. For the evaluation instrument, 17 grades ranged 
from 73 to 97, a spread of 24 points, while for the general impression 
scores the range was only from 85 to 95, a spread of only ten points. 

This would seem to substantiate the point that whn evaluating a per- ' 
formance on each of 12 different points, it is necessary to give more 
thought to the evaluation, and even though the question asked by each in- 
dividual point to be considered is subjective in nature, a more realistic 
score will be obtained than by assigning a grade based on an overall im- 
pression. 

A study of the difference between the grades a student gave those he 
"knows best" compared to those he "knows least" shows .that the students 
do grade those they "know best" higher than those they "know least." 
Of 17 students. 13 gave higher grades to their "friends," the difference 
ranging from one point to 11 points higher; two showed no difference in' 
their average grades; and two graded their "friends" lower by one point 
and five points. The average difference for the 17 students was a 
negative 3.4 points when using the evaluation instrument.; 

ERIC 15 



A siitiilar condition exists when using the general impression scores. Of 
17 students, 14 gave higher grades to their "friends," the difference 
being from one point to seven points; one graded both the same; and two 
graded their "friends" lower by one point and two points, respectively. 
The average difference was a negative 2.9 points for the general impre^s- 
v54on scores.- • 

This is less than the negative 3.4 points average difference from the 
evaluation instrument scores and was anticipated due to the narrower 
range of grades. . 
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ronclusion and Recommendations 

This research- tends to prove that when making a subjective evaluation, 
students grade those they know best higher than those they do not know 
''as well. While this bias is relatively high for some individuals, for 
the class as a whole the difference is only negative 3.4 points. A 
previous, less comprehensive study showed a similar bias of negative 2.3 
points. This bias is large enough to affect a student's grade but 
when averaged with the instructor's grade, the bias will be reduced by 
fifty percent as shown below. 

The average grade from all of the students' evaluations was 87.5 and the 
average difference between those they know best and those they know leas 
was a negative 3.4 points. The instructor's average grade for the same 
presentations was 84.1 points. Averaging these will give the following 
results : 



Students' Evaluation 
Instructors' Evaluation 

Average 

The study also indicated that a more realistic evaluation can be obtai 
by using an instrument requiring the evaluation of several different 
points instead of basing a grade on a general overall impression. A 
wider range of grades as well as a slightly lower average should be 
obtained by this method. 

17 



Know Most Know Least Difference 

87.5 84.1 -3.4 

84.1 84.1 

17V.6'" 168.2 

85.8 84.1 -1.7 



When an instructor wants assistance from others in grading a perrormance 
requiring subjective evaluation, this research shows that the instructor 
can use evaluations by other students in t\}e class. If an effective 
evaluation instrument is prepared and the Itudent^;' evaluations are 

averaged with those of the instructor, any personal bias in the students' 

11 ,.i ■ - . 

evaluations will be minimuzed. 



Student evaluations not only provide additional input to the instructor 
but givevthe students a greater sense of involvment and'lTarticiaption 

- \ 

^ in the class. 
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Exhibit One 



Purchasinq Negotiation 
104-192 

Evaluation of Purchasing Negotiation Presentation Date 

Fill in the names of the participants and sign your name. Circle, the rating 
number you think applies to each criteria, one being the lowest rating. All 
evaluation scores will be tabulated and tirade determined according to percentile 
rank. All evaluations- will be kept confidential. 



\ 



Names '> 




Buyer 














Seller 




1 


Criteria 




































How well were they prepared? 


■ 10 


9 


8 


7 


6 


5 4 


3 


2 






10 




3 


7 


6 


5 4 


t 

321' 

i 


To what degree were they in 
control of the situation? 


10 


9 


8 


7 


6 


5 4 


3 


2 


1 




10 


9 


8 




6 


b 4 


^ 1 1 


How successful were they i.n 
establishing a mocd favorable 
to .agreement?. 


10 


9 


8 


-• 


6 


S 4 


3 


2 


1 


10 


9 


8 


7 


6 




i 

0 c i 


How effective were the questions 
asked? 


10 


9 


8 


7 


6 


5 4 


3 


2 


1 




10 


9 


8 


7 


6 


5 4 


3 2 1 


How well did they respond to 
the questions? 


■" , 




8 


7 


6 


5 4 


3 


2 


.1 








8 


7 


6 


5 4 


3 2 r 


To what degre^e were alternate' 
solutions proposed? 






8 


7 


6 


5 4 


3 


2 


■ 








8 


7 


6 


5 4 


How well were they able to keep 
negotiations open and avoid 
a stalemate? 






8 


7 


6 


5 4 


3 


2 


ii 

1 \ 

\ 






8 


7 


6 

r 


5 4 


3 Ii 1 


To what extent was there com- 
plete understanding of the 
final terms and conditions? 












5 4 


3 


2 


. 

1 ; 














5 4 


3 2 1 


Hew effective were they in ex- 
ploiting their strong points? 


J" 




8 


7 


6 


5 4 


3 


2 


1 

1 








8 


7 


6 


5 4 


3 2 1 


How effective were they in con- 
cealing their weaknesses? 






8 


7 


6 


5 4 


3 


2 


1 








8 


7 


6 


5 4 


3 2 1 


How acceptable was their dress, 
speech and grooming? 












5 4' 


3 


2 


1 














5 4 


3 2 1- 


How well did the accomplish 


10 


9 


8 


7 


6 


5 4 


3 


2 


1 




'10 


9 


8 


7 


6 


5 4 


3 2 1 



their objective? 



.. ExhVjt T«io 



A 



- — 






1 






































None 

.I'll ' ■■"» i II 


How many hours 
do you spend with 

^hii per week out- 
•side of class? 






















- 
























Less than 1 hour 














































1 to 5 hours 






- 


\ 






































Hore .than 5 hours 


1 
j 














































i ! 











































Less than 6 months 


1 ... 
[How long have 

you known him? 

L 






















1 






£ .... 


















(6 months to ! year 


ii — 




































\ 








1 to 2 years 












'i 
































More than 2 years 








— 



















ft ^ 












4 






















































h very close frien 


K n 

*^How well are • 

1 














































A friend 






1 




















1 




















jyou acquainted 

Only as classmfftei ^ 
L — " jwith him? 
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Exhibit Three 
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ANALYSIS SHESr 



mean of scores given each student 



■ Lv^luator'i '.'lo. 
I — I.O. ''los. oi' ctudentS'ev3l';ytod 



variance from mean of evaluator's scores 



.factor from survey in-^'icating how well eval'ator 
is acquanitod with the stu('ents. '^k three ^ 
highest and lowest factors are listed. 



-Scores froni evciluatlon liistrujp,Gnt ' in 



in General Iri^rers^on srorfis 
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5" 
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z 
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.1,1.. 

^1 
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Exhibit Four 



SUMMARY OF ANALYSIS SHEETS 



Evaluation General 

Instrument ^Impression Difference 
Scores Scores. Between 



I. D. No. 


mi 


1 


mo 


d2 


^2 ~ ™1 


8837 


97 


- 5 


95 


- 5 


- 2 


6310 


95 . 


0 


93 


- 4 


- 2 


i 

8546 


92 


_ 4 


94 


- 3 


+ 2 ■ 


3429 


92 


-11 


93 


- 5 


+ 1 


1484 


91 


+ 1 


92 


- 1 


+ 1 


' 2365 


89 


- 9 


91 


- 5 


• + 2 


2910 


89 


- 5 


92 


- 5 


+ 3 


9223 


89 


_ 9 


92 


- 3 


+• 3 


0052 


88 


- 3 


94 


- 3 


+ 6 


3735 


O A.^ 


- z. 




_ 3 


+ 5 


5979 


87 


- 1 


90 


- 2 


+ 3 


5080 


. 86 


- 6 


85 


- 5 


- 1 


0972 


86 


0 


93 


+ 1 


+ 7 


8.434 


85 


-'>3_ 


91 


- 2 


+ 6 


9592 


83 


- 6 


88 


- 7 


+ 5 / 


6147 


79 


+ "5 


88 


+ 2 


+ 9 


6678 


73 


0 


87 


0 


• +14 


m 


87.5 


-3 .4 


91.1 


-2,9 


+3 .6 



Instructor 84.1 
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Exhibit Five 

-( ■ 

SUMMARY OF SCORES FROM EVALUATION INSTRUMENT 



I.D. 

NO. 


1st 
n_ 


Sess. 

m 


■ 2nd 
n_ 


Sess. 

ir 


Totals 
n_ 


m 


3566 


18 


79 


18 


8,6 


36 


83 


6310 


18 


93 


18 


96 


36 


95 


9223 


19 


•' 89 




89 
80 


24 


89 


9592 


16 ^ 


85 


10 


26 


83 


6632 


12 


78 


6 


"90 


18 


82 


2365 


19 


•86 


1.6 


91 


30 


89 


8546. 


20 


90 


18 


94 


38 


92 


2910 ' 


18 


89 


18 


93 


36 


89 


0972 -• 


16 


8Z 


18 


89 


34 ■ 


86-^ 


5080 


, 12 
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Explanation - 

•On the first session, student no. 3566 evaluated 18 presentations 
. - ' and the meanof his evaluations was 79, for the second session 18 
' O and 86 and the- total 3,6 and 83 respectively. 
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'Exhibit Six 

GENERAL IMPRESSION 
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Explanation - 

On the first session, student no. 3566 scored 18 presentations and 
the mean of his scores was 94.^;iFor the second session, 16 
and 93 and the total 34 and 94 ^respectively. 
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•X. . Exhibit Seven a 
SUMMARY OF RESPONSES FROM STUDENT SURVEY 
How many hours per week do you spend with him outside of class? 

Less Than More Than 



I.O. No. 
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Explanation - 

Student No. 3735 checked responses as follows regarding how many hours 
per week he spent with the other students outside of class.. 

None 7 

Less than 1 hour 3 

1 to 5 hours 10 

More than 5 hours 1 



Exhibit Seven b 
SUMMARY OF RESPONSES FROM STUDENT SURVEYS 
How Long Have You Known Him? 



I.D. # 


Less Than 
6 Mos. 


6 Mos- 
1 Yr. 


1-2 Yrs, 


More Than 
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Explanation - 

Student No. 3735 checked responses aT l^i^Uows^e'gardlng how long 
he had known the other students. 

Less than 6 months 0 

6 mos. to a yr. J 

1 yr. to 2 yrs. 12 

More than 2 yrs. 2 



Exhibit Seven c 



SUMMARY OF RESPONSES FROM STUDENT SURVEY 
How well are you acquainted with him? 



I. D. No. 


CI ose 
Friend 
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Classmate 
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Explanation - 

Student No. 3735 checked responses as follows regarding 
how well he was acquainted with the other students. 

Close Friend 1 
Friend 13 
Classmate 7 
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